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LONG-TERM  GOALS 

The  overall  goal  of  the  Electro-Optic  Identification  (EOID)  Research  Program  is  to  support  the 
performance  of  EOID  sensors  transitioning  to  the  fleet.  EOID  is  used  in  the  identification  of  Mine 
Like  Objects  (MLOs)  and  is  a  pressing  need  for  Mine  Countermeasures  (MCM)  operations.  The  EOID 
sensors  include  the  Streak  Tube  Imaging  LIDAR  (STIL),  which  is  transitioning  to  the  AN/AQS-20A 
and  the  WLD-1  (Remote  Mine-hunting  System)  programs,  and  the  Laser  Line  Scan  (LLS),  which  has 
been  delivered  to  the  Fleet  in  the  form  of  the  AN/AQS-14A(V1)  program.  Through  these  transitions, 
EOID  will  be  a  key  element  in  implementation  of  Fleet  plans  for  a  robust  organic  MCM  capability. 

The  EOID  Research  Program  will  begin  to  provide  the  tools  to  meet  specific  Fleet  needs  and 
capabilities,  which  include: 

•  Perform  mission  planning,  real-time  performance  assessment,  and  post-mission  analysis 

•  Flow  down  Fleet  identification  requirements  to  the  system  and  operational  parameters 

•  Develop  Computer  Aided  Identification  (CAI)  algorithms  to  aid  in  the  operator  identification  of 
mines 

•  Develop  Autonomous  identification  capability  for  future  systems 

•  Assess  and  evaluate  alternate  designs  for  future  systems 

OBJECTIVES 

The  primary  objective  of  this  phase  of  the  program  is  to  validate  existing  performance  prediction  and 
simulation  models  and  to  develop  and  test  Automatic  Target  Recognition  Algorithms  (ATR)  for 
electro-optic  identification  (EOID)  systems. 


1 


Form  Approved 
OMB  No.  0704-0188 


Report  Documentation  Page 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

30  SEP  2002 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2002  to  00-00-2002 

4.  TITLE  AND  SUBTITLE 

Electro-Optic  Identification  Research  Program:  Probability  of 
Identification  and  Performance  Modeling 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Coastal  Systems  Station„6703  W.  Highway  98„Panama  City„FL,  32407 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

The  overall  goal  of  the  Electro-Optic  Identification  (EOID)  Research  Program  is  to  support  the 
performance  of  EOID  sensors  transitioning  to  the  fleet.  EOID  is  used  in  the  identification  of  Mine  Like 
Objects  (MLOs)  and  is  a  pressing  need  for  Mine  Countermeasures  (MCM)  operations.  The  EOID  sensors 
include  the  Streak  Tube  Imaging  LIDAR  (STIL),  which  is  transitioning  to  the  AN/AQS-20A  and  the 
WLD-1  (Remote  Mine-hunting  System)  programs,  and  the  Laser  Line  Scan  (LLS),  which  has  been 
delivered  to  the  Fleet  in  the  form  of  the  AN/AQS-14A(V1)  program. 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

7 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


APPROACH 


To  have  a  true  performance  prediction  model,  two  components  must  exist.  There  must  first  be  a 
component  that  accurately  predicts  the  sensor  output  for  a  given  set  of  environmental  conditions. 

There  must  also  be  a  component  that  relates  the  sensor  output  to  real  performance  parameters  for  the 
Mine  Countermeasures  (MCM)  task.  This  program  dealt  with  both  components. 

The  performance  parameters  for  MCM  mission  are  Measures  of  Performance  (MOPs)  and  Measures  of 
Effectiveness  (MOEs).  The  two  Measures  of  Effectiveness  (MOEs)  for  MCM  operations  are  time  and 
residual  risk.  A  given  area  is  said  to  be  cleared  of  mines  to  a  certain  confidence  level  based  on  the 
search  effort  and  the  search  results.  A  more  confident  declaration  of  a  cleared  area  requires  more  time 
and  assets  to  be  applied  in  the  operation.  To  calculate  the  MOEs,  Measures  of  Performance  (MOPs) 
must  be  established  to  quantify  the  individual  system  contribution.  Some  of  these  MOPs  may  seem 
familiar  like  Probability  of  Detection,  others  like  Probability  of  False  Identification  (Pm)  and  Time  to 
Identify  (7jd)  may  not.  All  the  MOPs  must  be  defined  in  order  to  calculate  the  MOEs  of  time  and 
residual  risk.  To  get  to  the  MOPs,  it  is  necessary  to  relate  the  performance  back  to  sensor  and 
environmental  parameters.  The  goal  of  this  work  is  to  determine  Probability  of  Identification  (Pid)  as  a 
function  of  sensor  and  environmental  parameters. 

In  real  world  operations,  the  overall  MCM  MOE  is  sensitive  to  identification.  Search  systems  provide 
large  lists  of  mine-like  contacts  that  will  form  the  basis  of  prosecution  lists  for  identification  assets. 
Objects  positively  identified  as  mines  can  either  change  the  ship  operations  area  or  call  for 
neutralization  assets  to  clear  the  mined  area.  The  time  required  to  identify  the  objects  can  be 
significant. 

Performance  models  are  being  developed  so  operations  can  be  planned  using  water  clarity 
measurements  from  oceanographic  databases.  The  best  water  clarity  data  comes  from  sensors  in  the 
operation  area  during  the  operation.  The  Streak  Tube  Imaging  LIDAR  (STIL)  system  is  using  a  water 
clarity  measurement  to  suggest  an  operational  altitude  during  the  sortie.  Models  have  also  been 
suggested  to  show  reference  targets  as  they  would  be  seen  through  the  current  water  conditions. 

Ideally,  the  warfighter  would  be  able  take  whatever  information  is  known  of  the  environment,  input 
those  parameters  into  a  mission  planning  tool,  and  determine  the  expected  performance,  or  Pid,  to 
estimate  the  effectiveness  of  the  EOID  sensor  for  that  mission  on  that  day  with  that  target.  Figure  1 
illustrates  this  need. 
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Figure  1.  Mission  Planning  Tools.  [Block  diagram  of  mission  planning  tool  showing 
requirements  of  environmental  input  and  Measures  of  Performance  output] 

Currently,  there  does  not  exist  an  electro-optical  system  performance  model  for  the  design  and/or 
analysis  of  system  performance  in  the  application  of  mine  identification.  There  are  two  well- 
established  military  imaging  models:  target  acquisition  and  surveillance/reconnaissance.  The 
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surveillance  and  reconnaissance  model  approach  for  mine  identification  would  require  the 
development  of  a  National  Imagery  Interpretability  Rating  Scale  (NIIRS)  scale  for  mines  and  the 
development  of  a  general  image  quality  equation  (GIQE)  for  mines.  In  addition,  the  GIQE  has  four 
parameters  that  must  be  calibrated  for  the  equation.  In  contrast,  the  target  acquisition  model  has  been 
shown  to  be  successfully  applied  to  various  targets,  and  the  only  calibration  requirement  for 
application  to  underwater  mines  is  the  determination  of  a  fifty-percent  probability  of  identification 
cycle  criterion  (N50).  This  criterion,  coupled  with  a  Minimum  Resolvable  Contrast  (MRC)  model  for 
the  electro-optical  system,  allows  the  probability  of  mine  identification  to  be  calculated  for  any  electro- 
optical  system.  Therefore,  given  the  N50,  any  electro-optical  system  to  be  used  for  mine  identification 
can  be  evaluated  for  mine  identification  in  various  environmental  conditions. 

WORK  COMPLETED 

Two  tasks  were  completed  under  this  part  of  the  EOID  Research  Program.  First,  the  underwater 
imaging  model,  IMPERSonator,  was  completed  and  prepared  for  delivery.  Also,  using  the  contrast 
imagery  only,  an  experiment  was  conducted  to  determine  the  expected  Pm  performance  for  these 
sensors. 

The  primary  purpose  of  the  Plt|  experiment  was  to  determine  the  N50  requirement  for  the  identification 
of  underwater  mines.  The  identification  target  set  is  shown  in  Figure  2.  The  probability  of 
identification  performance  is  the  ability  of  an  observer  to  distinguish  one  of  these  targets  from  the 
other  eleven  targets. 

The  target  set  was  developed  at  NAVSEA  Coastal  Systems  Station,  Panama  City.  The  target  set 
includes  12  targets  in  9  orientations  with  a  nadir  illumination  source  (obliques  and  cardinals).  The 
targets  were  chosen  for  their  relative  confusability  and  tactical  significance.  Degradation  such  as  blur 
was  applied  to  these  images  to  simulate  various  sensor  and  environmental  effects.  The  mine  and  non¬ 
mine  target  images  were  developed  using  3  dimensional  models  that  were  textured  and  inserted  onto  a 
real  background  image.  For  the  experiment,  the  images  were  converted  to  12-bit  grayscale.  All  image 
sizes  were  400  x  400  pixels.  The  target  images  shown  were  processed  with  various  levels  of  blur. 


Figure  2.  Mine  Target  Identification  Set.  [Sample  target  set  includes  cylinder  shapes, 
truncated  cones,  and  trapezoidal  shapes  against  a  sand-rippled  background] 
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The  experimental  design  is  outlined  in  Table  1.  From  the  pristine  image  sets  of  Figure  2,  12  targets 
with  9  orientations  (108  images  total)  were  distributed  evenly  across  the  cells  in  the  table  shown.  The 
targets  were  then  distributed  across  the  cells  a  second  time.  This  distribution  allowed  for  three  of  the 
same  target  image  in  each  cell  and  four  different  targets  of  the  same  aspect  in  each  cell  resulting  in  36 
pristine  images  associated  with  each  cell.  The  36  images  were  processed  with  a  prescribed  blur.  In  the 
column  labeled  “10,”  36  pristine  target  images  were  blurred  with  a  10-pixel  blur  and  placed  in  cell 
“A.”  The  equation  for  the  blur  was  where  b  is  the  blur  in  display  pixels. 

1  -*(-)2 

h(x)  =  -e  »  (1) 

b 

This  Gaussian  blur  h(x)  was  to  be  applied  to  the  image  by  a  convolution.  The  total  blur  presented  to 
the  observer  included  a  combination  of  the  blur  given  in  equation  1,  the  blur  of  the  display,  and  the 
contrast  threshold  function  of  the  observer. 
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Table  1.  Experimental  Design.  [Experimental  design  showing  the  cells 
(A-F)  and  the  accompanying  pixel  blur  number  (10-60)] 

After  blurring,  the  images  were  combined  into  one  set  of  216  total  images  for  the  experiment. 
Observers  were  trained  to  identify  targets  with  a  minimum  95%  proficiency  (for  unblurred  images) 
prior  to  participating  in  the  experiment.  Each  observer  attempted  to  identify  all  216  images.  The 
images  were  randomized  in  order  to  vary  the  level  of  target  identification  difficulty.  The  image  sets 
had  equivalent  contrast  and  similar  size  characteristics.  The  contrast  was  described  with  the  root  sum 
squared  (RSS)  metric 


2Avgsc 


(2) 


where  crtgt  is  the  target  signal  standard  deviation  (within  the  target),  AG  is  the  difference  in  average 
signal  between  the  target  and  the  background,  and  Avgsc  is  the  average  scene  signal.  The  target 
characteristic  dimension  in  display  pixels  is  the  square  root  of  the  target  area. 

RESULTS 

The  experiment  was  conducted  at  NSWCDD  Coastal  Systems  Station  during  the  month  of  July  2002. 
Thirteen  observers  were  comprised  of  U.S.  Navy  Personnel  and  civilian  government  employees.  The 
results  were  reduced  for  an  ensemble  of  observers  against  an  ensemble  of  targets.  The  probabilities  of 
identification  were  calculated  with  all  observers  against  all  targets  within  a  cell.  The  experiment  was  a 
forced  choice  experiment  where  the  observers  had  to  choose  an  identification  choice  out  of  12  targets. 
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The  probability  results  were  averaged  across  all  observers  and  corrected  for  chance.  The  standard 
error  was  calculated  for  each  of  the  blurs,  and  the  probabilities  and  error  bars  were  plotted.  The 
limiting  frequency  was  then  calculated  for  the  human  contrast  threshold  function,  the  blur,  and  the 
monitor  for  the  given  monitor  luminance  and  the  target  contrast.  The  limiting  frequency  was  used  to 
determine  the  number  of  resolvable  cycles  across  the  target  dimension,  and  the  corrected  probabilities 
were  plotted  as  a  function  of  resolvable  cycles  on  target. 


p(id) 


-p(id) 

-TTPF 


Figure  3.  Probability  of  Mine  Identification  as  a  Function  of  Cycles  on  Target. 

[  Graph: probability  is  0.4  with  2  cycles,  0.85  with  6  cycles,  and  0.92  with  10  cycles] 


The  target  transfer  probability  function  (TTPF)  is  plotted  in  Figure  3  to  show  the  model  fit  to  the 
experimental  data.  The  form  of  the  TTPF  was  taken  as 


P,n(N)  = 


N 
V 


1.73 


l  +  (— )L7 

at 
1  v 


(3) 


where  this  form  has  been  shown  numerous  times  to  correspond  to  various  target  acquisition  observer 
responses  (tanks,  trucks,  humans,  handheld  objects,  etc.).  Nso  was  the  number  of  resolved  cycles  on 
target  for  a  fifty-percent  probability  of  mine  identification.  The  Nso  fit  for  the  mine  experimental  data 
corresponded  to  2. 1  resolvable  cycles  on  target.  Note  that  for  a  90  percent  probability  of  mine 
identification,  around  8.5  cycles  on  target  are  required.  Both  50  percent  probability  of  identification 
and  90  percent  probability  of  identification  are  common  specification  for  target  acquisition  systems. 

IMPACT/APPLICATIONS 


The  Navy  now  has  a  preliminary  baseline  function  for  calculating  P;d  both  from  model  output  and 
contrast  and  resolution  measurements  taken  from  actual  EOID  images.  The  function  was  derived  for 
contrast  images  only.  This  is  applicable  for  AN/AQS-14A  (VI)  and  the  contrast  portion  of  AN/AQS- 


20A. 


A  continuation  of  this  study  must  be  done  to  cover  the  range  imagery  and  combination  of  contrast  and 
range  imagery.  P;d  should  increase  with  the  additional  information.  Since  range  and  contrast  images  are 
not  truly  independent,  the  amount  of  gain  is  not  well  understood.  There  are  two  areas  that  experience 
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tells  us  range  will  be  a  major  contributor  to  Pid.  The  first  is  in  low  to  zero  contrast  situations  where 
there  is  little  information  in  the  contrast  image,  but  the  range  image  is  preserved.  The  second  situation 
is  differentiating  the  tire,  rock,  and  truncated  cone.  The  range  images  are  easier  to  determine  because 
the  targets  are  different  in  range,  but  very  similar  in  contrast. 

TRANSITIONS 

Pid  function  will  be  used  in  system  engineering  trade  studies  and  performance  prediction  models  for 
both  the  AN/AQS-14A  (VI)  and  AN/AQS-20A.  Additionally,  the  target  set  generated  for  the  study 
may  be  used  in  operator  mine  recognition  training.  Performance  models  for  the  Fleet  systems  are  still 
in  development,  but  the  Pid  function  is  a  critical  last  step  to  any  performance  estimate  since  the 
warfighter  does  not  consider  Signal  to  Noise  Ratio  or  resolution  a  measure  of  performance. 

RELATED  PROJECTS 

This  P id  function  is  based  on  a  simple  display  and  a  human  operator  making  the  call  using  the  image 
only.  As  advances  are  made  in  Computer  Aided  Identification  and  display  techniques,  the  Pid  tests  can 
be  rerun  to  quantify  the  change  in  performance  against  this  baseline. 
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